Systems and methods for detection, analysis, isolation and/or harvesting of biological objects

ABSTRACT

Systems and methods provide for detection and controlled interaction with one or more objects. The system can include an imaging subsystem (20), a tool subsystem (26) containing one or more tools, a stage subsystem (16) and a control system (40). The control system (40) can integrate controls for each of the other subsystems, which controls can be implement desired functions over a variety of process parameters to perform the controlled interaction.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.15/309,712, filed Nov. 8, 2016, and entitled SYSTEMS AND METHODS FORDETECTION, ANALYSIS, ISOLATION AND/OR HARVESTING OF BIOLOGICAL OBJECTS,which is a 371 of PCT App. Ser. No. PCT/US2015/029892, filed May 8,2015, and entitled SYSTEMS AND METHODS FOR DETECTION, ANALYSIS,ISOLATION AND/OR HARVESTING OF BIOLOGICAL OBJECTS, which claims thebenefit of priority from U.S. Provisional Patent Application No.61/990,387, filed May 8, 2014, and entitled AUTOMATED SYSTEM AND METHODFOR DETECTION, ISOLATION AND HARVESTING OF BIOLOGICAL OBJECTS. Each ofthe above-identified applications is incorporated herein by reference inits entirety.

BACKGROUND

Systems exist to provide for the automatic selection and harvesting ofcells and cell colonies. Such systems involve various levels ofautomation. Such systems can also utilize current selection algorithmsto facilitate the screening and harvesting of clone cells for a widerange of cell types and development of cell lines. While advancementscontinue to be made in such systems, further automation and improvedaccuracy is desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example of a block diagram of a system to facilitatedetection, isolation and harvesting of biological objects.

FIG. 2 depicts a plan view of an example of part of a system showingspatial relationships between a movable tip and a stage.

FIG. 3 depicts the system of FIG. 2 with the stage in a position forpart of a calibration process.

FIG. 4 depicts the system of FIG. 2 showing the stage in anotherlocation associated with the calibration process.

FIG. 5 depicts the system of FIG. 2 with the tool arm moved to alocation for receiving a new tip.

FIG. 6 depicts an example of the system of FIG. 2 demonstrating part ofa tip sensing process.

FIG. 7 depicts an example of the system of FIG. 2 demonstrating anotherpart of the tip sensing process of FIG. 6 .

FIG. 8 depicts an example of a non contact tip sensing system that canbe utilized for detecting the tip location with respect to axes of thestage.

FIGS. 9 and 10 depict examples of a Z-axis sensor to detect tip heightalong the Z axis.

FIG. 11 depicts an example of a topographical map of a surface that canbe detected using the tip height sensing of FIGS. 9 and 10 .

FIG. 12 is a flow diagram depicting an example of a calibration method.

FIG. 13 depicts an example of a method for interacting with an objectidentified in an image.

FIG. 14 depicts an example of a method to determine the tip height inassociation with and identifying a location of an image to enableinteraction with the object.

FIG. 15 depicts examples of images of objects on a stage associated witha sequence of interactions for transferring identified biologicalobjects.

FIGS. 16A and 16B depict an example of another harvest sequence ofinteractions that can be implemented.

FIG. 17 depicts example of part of an interaction protocol forharvesting biological objects that is updated dynamically based onpost-interaction image data.

DETAILED DESCRIPTION

This disclosure relates to automated systems and methods for detection,sequential analysis, isolation and/or harvesting of biological objects.The system can include an imaging subsystem, a tool subsystem, a stagesubsystem and a control system. The control system can integratecontrols for each of the other subsystems, which controls can beinterdependent to implement desired functions over a variety of processparameters.

As an example, a method can include identifying at least one object ofinterest based on image data representing the object of interestresiding in medium, such as acquired by the imaging system. An imageanalysis function, which can be part of the control system or includeseparate analytics, can be configured to analyze the acquired image datato determine a distribution of pixels or voxels of an image thatcorresponds to the object of interest. The control system can controlone or more forms of interaction with the object of interest based onthe distribution of pixels or voxels. The control can further be updateddynamically during the interaction based on image data acquired duringsuch interaction to enable corresponding adjustments to one or more ofthe controls.

The tool system may include tools that are used to interact with themedium or interact directly with the biological object. The interactionwith the medium may include addition or removal of medium elementsresulting in a change in composition of the medium. The interaction withthe medium may also include mechanical or biophysical modification ofthe medium conditions such as through movement, agitation, heating,cooling or application of external biophysical stimuli (e.g. lightenergy or electromagnetic fields). Interaction can also include use of atool having an interior space through which fluid or medium may flow,and the use of this tool to locate and aspirating medium using a fluidicsystem. Interaction, including but not limited to aspiration, may beused to sample, harvest, move, remove or kill biological objects (e.g.,cells or colonies of cells) or other material that may complicate orhinder the effectiveness of the interaction with the object. As anotherexample, the interaction can include acquiring one or more images, whichcan be analyzed to determine information and process parameters. Imagestaken at two points in time may also be used to detect and measurechanges that have taken place in the biological object or objects as aresult of the interaction.

FIG. 1 depicts a block diagram of an example of an automated system 10for detection, sequential analysis, isolation and harvesting of objects.As used herein, the object being harvested can correspond to a cell, agroup of cells including a colony or a collection of cells correspondingto a population of cells, or cells forming a tissue. While many examplesherein describe the objects as including cells or colonies of cells, itis to be understood and appreciated that the system and methods are notlimited in application to cells as images of other types of objects canalso be processed according to an aspect of the present invention. Forexample, the objects can also include microorganisms (e.g., bacteria,protozoa), cell nuclei, cell organelles, viruses, non-biologicalstructures (e.g., proteins, peptides or glycoproteins), differentconstituent parts of such objects or any combination of one or more ofsuch objects.

In order to control process parameters, the system 10 includes a controlsystem 40 that is programmed to control the various subsystems of thesystem including a stage motion system 16, imaging system 20, and theautomated tool system 26 (e.g., further including tool function system30 and tool motion system 28). Thus, in the example of FIG. 1 , thecontrol system 40 includes an imaging control 42, stage motion control44, tool motion control 46 and tool function control 48. The controlsystem 40 provides an integrated control method to coordinate control ofthe various system components 20, 26 and 16 to perform a variety offunctions, as disclosed herein. While in the example of FIG. 1 forpurposes of explanation each of the respective control blocks 42, 44, 46and 48 are demonstrated as separate modules (e.g., program code blocksexecutable by a processor), it is to be understood that the methods andfunctions implemented by such controls can be integrated andinterdependent to effect desired isolation detection and harvesting ofbiological objects.

In the example of FIG. 1 , the system 10 includes a stage 12 thatsupports the objects of interest 14. The position and movement of thestage 12 in a defined stage coordinate system can be controlled by astage motion system 16. The stage motion system 16 can include anarrangement of motors connected to drive the stage 12 to a desiredposition in accordance system based on profits parameter. For example,the stage motion system 16 can include an arrangement of linear motorsconfigured to adjust the position of the stage 12 along two or more(e.g., three) orthogonal axes, corresponding to X and Y axes.

The objects 14 that are positioned on the stage can correspond to platedcells residing in a known medium, such as can be a liquid medium or aviscous or solid (e.g., alginate, methycellulose, hyalyonan, or otherhydrogel composed of natural or synthetic polymeric gel materials or thelike). One or more types of marker criteria can be utilized todifferentiate or optically label structures and/or chemical features ofobjects in a given image. Examples of some marker criteria includestaining (e.g., with one or more dyes), employing phase contrastmicroscopy, topographic image mapping of a two-dimensional surface.Additionally or alternatively, for various types of cells the markercriteria can include an immunochemical marker, a histochemical marker,as well as an in situ hybridization marker. Those skilled in the artwill understand and appreciate particular applications of these andother types of marker criteria that can be used to optically identifyobjects and/or chemical features of objects in an image.

By way of example, AP staining can be employed as a mechanism to analyzeundifferentiated progenitor cells, for example. AP and DAPI stains canbe utilized together on a given sample, such as for locating cells(e.g., via DAPI or other staining) as well as analyzing performancecharacteristics of certain types of cells (e.g., via AP or otherstaining). It is to be appreciated that other stains or dyes can also beutilized in addition to or as an alternative to AP and/or DAPI staining.The particular types of staining employed can vary according to thetypes of objects being selectively stained. For example, specific typesof cells, different components within cells (e.g., organelles, proteinsor other molecules) as well as other objects can be selectively stainedand provided as the one or more objects of interest.

Markers can also be employed to identify (e.g., via image analysis 24)chemical features or morphologic features in the matrix materials aroundand near the cells, which further can be used to characterize and assessbiological identity or performance of the adjacent cells. For example,chemical features may include proteins or other chemical compounds thatmay be secreted or deposited by cells. Morphologic features near thecells can include supercellular features (e.g., collections of cellsinto geometric structures, such as tubular and rosette shapes), minerals(e.g., calcium-based compounds) formed near the cells, fibrous proteinsformed near cells, as well as the size and configuration of junctionpoints between cells to name a few. Other morphological features caninclude physical features of cells or groups of cells, such as includingsize, shape, optical density, auto-fluorescence, presence of absence ofcell surface markers, presence or absence of specific extracellularmatrix components and/or presence of specific enzymatic activity.Additionally, groups of cells can be identified as functionally relatedto one another as a biologically relevant group (e.g., a group of cellslikely sharing a common ancestral cells (colonies), or a group of cellsresponding (or not) to a specific signal, cells meeting (or not meeting)metrics defining a desired range or constellation of features).

The medium can be a standard medium or it can be user selected andinformation about the medium (e.g., material properties, opticalproperties and the like) can be entered into the system 10 via acorresponding user interface 18. The user interface 18 can be programmedto provide a human machine interface through which one or more users caninteract with the system 10. Such human-machine interactions by the usercan include setting operating parameters and thresholds utilized by thesystem. The human-machine interactions can also include remotelycontrolling the positioning of the stage along one or more of its axes.The user interface 18 can also be employed to define properties,interaction protocols, and/or process parameters associated with theobjects, the medium for the objects or other parameters or criteria thatcan be utilized in conjunction with the detection isolation and/orharvesting of the objects from the stage 12.

The imaging system 20 is configured to acquire image data 22 thatincludes one or more images collected from the stage. As used herein,the images can be static images captured at an instantaneous time and/orvideo images captured over a time interval. By way of example, theimaging system 20 can include a digital camera 60, such as can includean arrangement of one or more digital sensors, analog sensors, chargecoupled device (CCD) sensor, complementary metal oxide semiconductor(CMOS) sensor, charge injection device (CID) sensor. The arrangement ofoptical sensors may be implemented as a two-dimensional array. The oneor more sensors, for instance, can be implemented in a digital camera 60or the sensors could be a special purpose imaging device. The imagingsystem 20 provides an output signal corresponding to image data inresponse to detecting light (e.g., visible and/or non-visible light).The imaging system 20 can be configured to operate in an automatedmanner to acquire images of the stage and store corresponding image dataover a period of time.

In the example of FIG. 1 , the imaging system 20 is demonstrated as alsoincluding optics 58 that can control the field of view and resolution.The imaging system can also include one or more lights and/or filters 62that can be utilized to change the type of illumination and/or filtersapplied for removing selected wavelengths of the illuminated light fromthe image being captured by the camera 60. The optics 58 can be selectedto refine an object field of view but in turn can be passed to thecamera 60 for capturing the corresponding digital image.

For example, the camera 60 can be implemented as a digital camera thatcan be attached to a microscope containing the optics 58 for capturingat least a portion of the field of view as pixels having values thatcollectively form a corresponding digital image that is stored as theimage data 22. For each image that is stored in the image data 22, thecontrol system 40 can include metadata with the image data specifyingparameters of the imaging system 20, such as image resolution, time anddate information, the associated optics setting(s) as well as anindication of the light source and/or filtering that is utilized for thecaptured image.

Additionally, the control system 40 can provide location information inthe metadata 23 that represents other state parameters for the system,including a spatial location (e.g., in stage coordinates in two or moredimensions) for each captured image. For example, the location metadatafor each image can represent the spatial coordinates of the stage at atime when each respective image is acquired, such as corresponding to orderived from linear position encoders that provide absolution positionfor the stage along its respective axes. Additionally or alternatively,the location coordinates can be stored in the metadata 23 to identifythe spatial position in stage coordinates that has been converted to animage coordinate for one or more predetermined pixels (e.g., at acenter, along a perimeter or other locations) in the captured image,such as by registering the spatial coordinates of the stage to apredetermined location or other marker on the stage. Since the spatiallocation of the stage is known for each image (e.g., from absoluteencoder data provided by respective encoders in response to position ofthe stage) and an offset between the tool and at least one pixel in thefield of view is also known with respect to stage coordinates, asdisclosed herein, the stage can be moved relative to the tool, toaccurately position the tool in alignment for interaction with an objectin each respective image. For example, given a set of pixels of interestcorresponding to a target object, the spatial offset between the tip ofa tool and optical system can be employed to move the stage relative tothe tip to position the tip in axial alignment with the target object.Other machine state information that can be part of the metadata 23 caninclude time, current position of all monitor components (e.g., fromrespective position encoders).

As a further example, the imaging system 20 can be configured to acquirea temporal sequence of images for a localized set of one or morecells/colonies at defined time intervals. Such serially acquired imagescan be used to identify changes that occur between two or more serialimages. The identified changes can be interpreted as biological eventsof functional significance (e.g., proliferation, migration, change inphysical, chemical, anabolic, catabolic or secretory properties) at thelevel of an individual cell or at the colony level, for example.

The image data 22 can include values for pixel or voxels in each imageas well as process related data and/or metadata, such as timestamps,temperature, pressure, medium conditions, and the like. An imageanalysis module 24 can be programmed to apply preprogrammed analytics toanalyze and evaluate the imaging data that is acquired to enabledetection and isolation of the biological objects of interest. Anexample of an image analysis that can be utilized is shown and describedin U.S. Pat. No. 8,068,670, which is incorporated herein by reference.The '670 patent also discloses an approach for stitching a spatiallycontiguous sequence of images together to create a corresponding montageimage that includes a plurality of adjacent field of views. Metadata,including spatial coordinates, can be stored with each of the individualimages that have been stitched together to form the correspondingmontage image to facilitate interaction with objects that may beidentified at different parts of the montage image.

The pixels or voxels for a given image can represent imaging dataacquired by the imaging system 20 from one or a plurality of imagingdomains or imaging modalities (e.g. bright field microscopy, phasecontrast, Numarski imaging, fluorescence microscopy, or the like). Suchmultimodal imaging parameters can enable changes in individual andcollective objects to be measured, for example, with respect totransmitted light, phase contrast, fluorescence, and other spectralfeatures. Thus, the image analysis 24 can employ a mathematicalalgorithm to integrate imaging data from one or more domains forcharacterization and interpretation of values in the pixels or voxels(e.g. a ratio of brightness or product of brightness). The imageanalysis 24 can also be configured to provide summary analysis reportsitemizing features of individual cells, groups of cells and/or therelationship of individual cells to defined groups within each of theacquired images. The resulting analysis data can also be stored as partof the image data 22 to facilitate further isolation and harvesting ofbiological objects.

As a further example, the image analysis 24 can be programmed toquantify changes at the level of individual biological objectsincluding, for example, with respect to attachment, migration, survival,metabolic activity, protein secretion, expression of surface markers,and/or morphological changes. Additionally or alternatively, the imageanalysis 24 can quantify changes at the level of the entire populationof biological objects with respect to attachment, migration, survival,metabolic activity, protein secretion, expression of surface markers,and/or morphological changes.

The system 10 can also include the automated tool system 26 that isconfigured to interact with one or more selected objects 14 based on atleast in part on the distribution of pixels or voxels that is part ofthe image data 22. The automated tool system 26 can be configured toposition one or more tools in a location for interacting with medium orone or more of the biological objects 14 disposed on or in such medium.The automated tool system 26 further can include a tool motion system 28that is configured to adjust the position of a tool or more than onetool relative to the stage 12. The tool motion system 28 can include anarrangement of linear motors, rotary motors and/or other types ofactuators configured to control movement of the syringe and componentsthat may be attached thereto, such as needles or various tip shapes, inthree-dimensional space. For example, an arrangement of linear motorscan be provided to enable precise positioning of the tool along threeorthogonal axes.

An example of a device that can be implemented as the tool system 26 orat least a portion thereof is disclosed in U.S. patent application Ser.No. 13/701,853 (U.S. Pat. Pub. No. 20130129538), filed Feb. 7, 2013, andentitled MINIATURIZED SYRINGE PUMP SYSTEM AND MODULES, which isincorporated herein by reference in its entirety. The tool motion system28 can move the tool, including a tip of such tool, along a z-axis thatis orthogonal to the surface of the stage to enable interaction betweenthe tip of the tool and the media and/or biological objects disposedthereon. Additionally or alternatively, the tool motion system 28 canmove the stage along X and/or Y axes such as to provide for positioningthe tool in three-dimensional space in response to control instructionsfrom the control system to the motion system 28.

As a further example, the control system 40 can provide instructions,corresponding to motion data 36, to the motion system 28 to move thetool to a predefined location in the coordinate system of thecorresponding tool having a predefined position (e.g., tool referencecoordinates) in the coordinate system of the stage 12 (or to some othercommon coordinate system), which can be utilized for accuratelypositioning the one or more tools with respect to a target locationidentified in one or more images. Calibration disclosed herein betweenthe tool system 26 and the image system 20 enables precise positioningof the tool with respective locations identified in captured images.

In order to enable such precise positioning of the tool, the controlsystem 40 can implement a calibration function 54 for the system 10. Forexample, the system calibration function 54 can determine a spatialoffset between a tip of a tool of the automated tool system 26 withrespect to an optical location within the field of view of the imagingsystem 20. The spatial offset can be stored in memory and utilized bythe control system 40 to position the stage 12, accurately andreproducibly, relative to the tip of the tool. As mentioned, the imagescan be stored in the image data 22 associated with correspondingmetadata 23, including position metadata. Since the metadata 23associated with each respective image can include a correspondingspatial position of the stage associated with one or more pixels thereof(e.g., a center pixel or a predetermined edge pixel), accuratecoordinates of an object identified with respect to one or more pixel inthe image can be readily determined with respect to the stage andthereby enable the control system 40 to adjust the position of the stageinto alignment with a desired tip of the automated tool system 26.

By way of example, the system calibration function 54 is programmed tocontrol the stage motion system 28 to move to a plate or other trayunder the tool tip that has been positioned to its predefined referencelocation and aligned with a location. An ink or other transferrablemedium (e.g., ink or other marker material), which is visible to theimaging system 20 can be applied to the tip of the tool. The indicia canbe applied to the stage or an object disposed thereon by moving the tipcarrying a marker material (e.g., ink or other marker material) intocontact with the surface of the stage, for example. The position of thestage (e.g., in two- or three dimensional space) can be recorded. Thisstage carrying the applied indicia is then moved under the microscopeuntil the applied marker is located and a new coordinate position of thestage (e.g., in two- or three dimensional space) is recorded. With thesetwo pieces of information, the relative location of the tool referenceposition and the microscope optical centerline is known. This tip tooptical offset information is used to calibrate the system to enableprecise and reproducible positioning of the tool with respect to targetson a corresponding set of one or more well plates identified in thecaptured images. The tip to optical offset calibration can be performedfor each set of plates or other apparatuses that are positioned on thestage.

In the addition to the system calibration 54 determining the tip tooptical offset, such system calibration 54 can also ascertain atip-to-tip spatial offset between a reference tip and one or more othertips that can replace the reference tip on the tool (e.g., a mandrel).As used herein, the reference tip can correspond to a given tip that isutilized initially to determine the tip to optical offset mentionedabove and each other tip that is attached to the tool (e.g., a mandrel,such as a hollow body of a syringe tool apparatus) can be calibrated todetermine a corresponding tip-to-tip spatial offset with respect to thesame given reference tip.

The system 10 can include one or more sensors integrated into the stage12 to provide tip position data representing a location of the tip withrespect to each of the orthogonal (e.g., X,Y) axis of the stage 12. Forexample, the tool can be positioned at its predefined reference position(e.g., based on predetermined tool motion data 36 stored in memory), andthe stage can move relative to the tip, while the tool is located at itspredefined reference position, to identify the coordinates of the tipwith respect to the axes of the stage based on the tip sensor data. Asdisclosed herein, the reference position of the tool can be anypredefined repeatable position to which the tool can be accuratelypositioned. This can be repeated for a subsequent tip and the differencebetween the tip locations, as provided by the tip sensors, can definethe tip-to-tip offset between the other tip and the reference tip. Thesystem calibration 54 of the control system can in turn aggregate thetip-to-tip offset with the tip to optical spatial offset determined forthe reference tip to enable accurate positioning of the stage 12 via thestage motion system 16 with respect to the new tip. Examples of such tipsensing and calibration functions are disclosed herein with respect toFIGS. 6-8 .

In some examples, the automated tool system 26 can also include a toolfunction system 30 that is configured to activate an associated functionof the tool for interacting with an identified target (e.g., one or morecells or cell colonies) based on tool action data 38. For the example,the tool comprises a syringe apparatus that includes a hollow bodymandrel to which a tip is removably attached and that provides fluidcommunication to a fluid source. In such example, the tool functionsystem 30 can activate the syringe for aspirating with the syringe bycontrolling the flow rate and volume of material that can be drawn intoa tip of a tool containing an inner channel through which a controlledflow of fluid or medium can be implemented to aspirate or dispense.

Examples of variables that can be selectively controlled based on theimage analysis 24 can include volume, flow rate, force, time and/or rateof change of pressure (e.g., positive or negative pressure), height ofthe tip relative to the object and/or stage 12 as well as pattern andmovement of the tip relative to the stage 12, as well as combinationsthereof. The relative motion between the tool tip and the stage can becontrolled, for example, based on the tool motion system 28 along two ormore of orthogonal axes. Alternatively or additionally, relative motionbetween the tool tip and the stage can be controlled based on the stagemotion system 16 controlling position of the stage along two or moreorthogonal axes. For example, the tool tip can be moved to a computedcentroid of the target object and the tip can be positioned at acomputed distance (e.g., along the Z axis) from the stage for aspirationor other interaction. The dimensions for a selected tip can be utilizedas interactive protocol and process parameter to enable control of thetool subsystem and the tool and fluidics system, for example.

As a further example, the fluid flow parameters can be stored in memory,such as part of tool action data 38 that includes a library ofinteraction protocols. Each respective interaction protocol can includea series of “process steps” or actions, each having a defined set of“parameters” that characterize each step that is to be performed in asequence or concurrently. The library of interaction protocols canincludes predetermined schemes applicable based on the sensed data andimage analysis. Additionally, the interaction protocols may include oneor more user configurable schemes that can be generated in response toinputs via the user interface 18. As one example, a given aspirationscheme can be selected as part of a given interaction protocol based onthe image data 22 (including derived from image analysis 24) and otherdata (e.g., sensor data 52, tool motion data 36, user input data and thelike).

For example, a given interaction protocol may be selected from a libraryof one or more pre-programmed interaction protocol, which can vary withrespect to variables associated with tool selection and tool aspirationcontrols. For example, the schemes can be programmed to vary based on,including but not limited to: tool selection, aspiration tool design androtational orientation, pattern of movement, height during aspiration,flow rate, volume aspirated, pattern of directionality of flow or toolmovement, as well as stage motion. The use of chelating agents orchanges in plate or fluid temperature to reduce cell-cell adhesion orcell surface adhesion may also be used. The addition of chelating agents(e.g. ethylene glycol tetraacetic acid—EGTA; Ethylenediaminetetraaceticacid—EDTA), changes in plate or fluid temperature, or addition of otherbioactive agents prior to aspiration may also be used to reducecell-cell adhesion or cell surface adhesion. A given interactionprotocol can be selected or modified based on object cell type (e.g.,connective tissue progenitor cells, mesenchymal stem cells, endothelialcell, keratinocyte, neuron, induced pluripotent stem cells, or thelike). The selected protocol may be updated or modified based on objectfeatures (e.g., size/area, size/cell number, thickness, density,distribution (e.g., uniform or dense center), substrate/surfacefeatures, extracellular matrix features or the like.

Data associated with the tool motion can be stored as part of the toolmotion data 36. Similarly, data associated with the aspiration,including aspiration control parameters and aspiration scheme, can bestored in memory as part of the tool action data 38. The tool motiondata 36 and the action data 38 further can provide feedback informationduring the process which can further be utilized to refine the variablesand process parameters in a selected interaction protocol. The feedbackcan be derived from multiple acquired images (e.g., over a timeinterval) of areas that have the site of tool interaction or fluid flowintervention or aspiration. Basically, after a cell colony is located,the control system 40 activates corresponding process controls 42, 44,46 and 48 to interact with the colony according to each sequence ofsteps in the selected interaction protocol. After the interaction, thesample containing the biological object is assessed using the automatedimaging processing (e.g., image analysis 24) of acquired image data 22.Analysis of images before and after interaction can then be used todetect and measure the effect of the interaction and/or the combinedeffect of a sequence of interactions. These data can then be used, basedon predetermined criteria, to proceed to a next phase in a series ofsteps in an automated interaction protocol. Alternatively, direct visualfeedback information can be provided derived based on a comparisonbetween image data before and after an interaction or series ofinteractions. From this visual feedback information compensations can bemanually entered in to the motion algorithm for a subsequent interactionwith the objects on the stage.

Additionally, during the process, image data 22 can be acquiredcontinuously or intermittently. For example, actions associated with themovement of the syringe can occur and corresponding image data can beacquired. The control system 40 can employ the acquired image data todynamically update and adjust the position parameters for the tool basedon the analysis of the image data 22. As a further example, thedistribution of pixels of voxels corresponding to the image data can beupdated dynamically during the process of tool motion and/or aspirationbased on the acquired image data to enable corresponding adjustments(e.g., in substantially real time) by the tool motion system and thetool function system 30.

The control system 40 further can access the data that is provided byone or more of the respective subsystems, including the image data 22,stage data 17, tool motion data 36 and tool action data 38. The system10 can also include one or more environmental sensors 50 that can beconfigured to provide corresponding sensor data 52. Environmentalsensors can include sensors for providing sensor data indicative oftemperature, environmental pressure in the handling chamber, humidityand the like. The control system 40 thus can also receive the sensordata 52 for implementing corresponding control of the system 10.

For example, the imaging control 42 can control various parameters ofthe imaging system 20, including to control which optics are employed aswell as activation of a light source and optical filters utilized duringimage acquisition. The imaging control 42 also controls activation ofthe camera acquires images (e.g., at automatic intervals and/or inresponse to user input via user interface 18). The imaging control 42further enables the image analysis 24 to analyze values of the pixels orvoxels in the corresponding distribution of pixels or voxels provided bythe image data 22.

As an example, the image analysis 24 can determine one or more featuresof an object of interest based on the values of pixels or voxels. Thenbased on the determined features, the tool motion control 46, stagemotion control 44 and/or function control 48 can provide correspondingcontrol signals to the automated fluidic tool system 26 to controlpositioning of the tool then implement desired interaction withidentified objects.

As another example, the tool function control 48 can be configured tocontrol one or more of the volumes of fluid to be aspirated as well asthe flow rate during aspiration. For instance, the function control 48can control the positive and negative pressure applied across the tipcan with respect to flow rate and direction, thereby controlling theflow rate and volume of material passing through the tip of an automatedsyringe apparatus that is implemented as the automated tool system 26.In some examples, the function control 48 can control the flow rate upto about 500 μl/second or greater.

Additionally or alternatively, as part of the process, the tool motioncontrol 46 and/or the stage motion control 44 can be configured toadjust a height of a tool tip relative to the surface containing theobject of interest and the medium and/or the articulating pattern of theneedle during aspiration, such as based on one or more of the acquireddata 17, 22, 36, 38 and 52. The interoperation between the subsystems isdriven mainly from the optical information obtained from image data 22and the image metadata 23. The location and size of the cells arecomputed (e.g., in system coordinates), such as disclosed herein,corresponding motion and aspiration processes are executed using one ofthe interaction protocols.

As yet a further example, the tip (e.g., needle of the syringe) can bereplaced or removed in an automated manner. For instance, the toolsystem can include a replaceable tip, which can be automaticallyselected from a plurality of different available tip designs. A selectedtip can be configured to attach a distal end of the tool via a matinginterface. Various types of interfaces (e.g., friction fitting, threadedinterface or the like) can be implemented. As one example, the interfacecan be a wedge shaped press fit on the tool that may be applied bypressing the fitting at the free distal end of the tool (e.g., amandrel) into the tip. The tool system 26 controls the pressure duringtip attachment, such as between two levels—high enough to make a solidinterface by not too high that it damages the tip. The tool of the toolsystem 26 can include an integrated force sensor in the bracket thatholds the tip. The force sensor can provide force information (e.g., aspart of tool action data 38), which is monitored by the control system40. The control system 40 can providing the force information back tothe tool motion system 28 to control the force in the Z-direction whilethe tip is being applied to the tool holder. The tool system 26 can alsoinclude a spring loaded holder that helps ensure proper force is appliedduring loading.

Different tip designs can be selected (e.g., by control system or inresponse to a user input) depending on the interaction that is neededand the medium from which the cells and/or colonies and nearby media areto be extracted. Different media can be categorized differently, such asa fluid, viscous fluid, semi-solid, gel, hydrogel or the like, and a tipcan be selected according to the categorization of the medium that ispresent. The category of medium can further be a process parameter usedto control aspiration for a given interaction protocol.

Additionally, the tools can be designed for single use and be disposedof in a receptacle after use. For a replaceable tip design, removal of atip may be implemented by a “shucking” device. For example, the shuckingdevice can be implemented as a keyhole structure (e.g., a large diameterhole intersected by a smaller diameter slot) located a predefined toolcoordinates. The loaded tool is lowered into the large hole and oncebelow inside the key hole, the tool is moved laterally into the smallerslot area. The tool is then moved up (e.g., in the Z direction) suchthat the tip catches the edge of the key hole slot and is removed fromthe tool. The key hole can be located such that the tip will drop into awaste receptacle. In other examples, another means (e.g., grippingdevice, such as a clamp or the like) can be used to remove the tip fromthe tool body.

In other examples, the tip may be reusable. For instance, the controlsystem 40 can be configured to implement automated washing and/orsterilization of the tool tip and reservoir between successiveinteractions with a given tool tip that may be integral with the tool(e.g., fixed as not intended to be replaceable). Examples of tools thatmay be fixed or otherwise not replaced may include a cutting tool, ascraping tool, a stamping tool or a stirring tool. In other examples,the tool system 26 can include a plurality of different tools that canbe utilized sequentially or concurrently to interact with selectedobjects, including aspiration of objects into respective tips. Some tipsfurther can be multi-purpose tips to perform more than one of theinteraction functions disclosed herein.

As a further example, an interaction can include pre-treatment appliedvia one or more tools, such as to prepare one or more target objects forsubsequent interaction. Examples of pre-treatment can include: removalof non-adherent cells or material (e.g. change of medium overlying theobject prior to interaction); removal of cells or material from theobject prior to interaction (e.g. blow off cells or material looselyadherent to the object); remove adjacent cells or material that maycomplicate or hinder the effectiveness of the interaction with theobject; application of pretreatment chemical or biophysical methods tochange the interaction of the object or object components (cells) witheach other or the underlying surface such as Examples of pre-treatmentchemical or biophysical methods may include medium agitation (e.g.Shaking or stirring), temperature Change within the chamber, mediaChange (e.g., removing Ca or Mg) and/or enzymatic digestion andquenching (e.g, with associated time controls).

Additional controls can also be implemented by the stage motion control42 and/or the tool motion control 46 for controlling the interaction ofthe object 14 located on the stage 12. As disclosed herein, for example,the control system 40 can be programmed to determine the distance of theobject of interest that is located in the medium on the stage 12relative to at least one other object located in the same medium basedon the distribution of pixels or voxels provided by the image data 22.For instance, the distance between objects can be based on the imageanalysis identifying objects and computing centroids for each object andthen computing a corresponding distance between the respectivecentroids. The distance can be an absolute distance or it can be adistance computed as a number of pixels or voxels along a line betweenrespective centroids, for example. In other examples, the position ofthe object of interest can be computed in a desired spatial coordinatesystem to facilitate interaction. The control system 40 can be furtherprogrammed to control the type of interaction or exclude interactionwith the given object based on the determined distance and otherconditions determined based on the image data and other processparameters. For instance, if an object is too close to another objectthe determination can be made to exclude aspirating the cells at sucharea.

In the event that an object of interest is in proximity to other objectsor materials of non-interest, means can be employed to separate theobject of interest from non-interest prior to direct interaction withthe object of interest by means of physical interaction using anautomated aspirating or non-aspirating tool to dislodge or removeadjacent or adherent objects or materials. For example, the tool actioncontrol can be programmed to direct a controlled flow of fluid over thetop of the object to displace less adherent cells or materials.Alternatively, an aspirating or non-aspirating tool may be manipulatedacross the surface to dislodge and remove objects of non-interest,leaving an object of interest effectively isolated for use or making itmore accessible for precise controlled interaction at a subsequent stepin the current interaction protocol.

The imaging analysis 24 can also be programmed to divide thedistribution of pixels detected from an image into discrete spatialregions and/or form a montage of plurality of discrete images, such asdisclosed in Appendix A. Each spatial region is assigned its ownrespective distribution of pixels or voxels therein. The distribution ofpixels or voxels for each respective spatial region can be utilized bythe control system 40 to control the automated tool (e.g., syringe) 26for performing selective aspiration on based on the distribution ofpixels or voxels within the given respected spatial region.Additionally, the tool function control 48 can be further programmed todetermine an interaction protocol that is selected for each discretespatial region based on the corresponding distribution of pixels orvoxels for each such region. In this way the tool function control 48can set a corresponding interaction protocol (e.g., for aspirationand/or another form of interaction), such as for setting the processingparameters and controlling aspiration of each object of interest that islocated in each respective spatial region.

As mentioned above, the imaging control 42 can be programmed to continueto acquire images either during interaction/aspiration or in betweendiscrete phases of the interaction process. For example, the imagingsystem 20 can update the image data to reflect changes in thedistribution of pixels or voxels for each of the respective spatialregions within an image field of view in response to correspondinginteractions (e.g., aspirations) with the objects of interest. In yetanother example, the imaging system can update the spatial region inwhich the object resides and the immediate adjacent neighboring regionsto the region containing the object of interest. The image analysisfunction 24 can further divide each of the images into correspondingdiscrete spatial regions based on location metadata that is embedded inthe image data 22. Such spatial regions may be the same or varythroughout the interaction process. The image analysis 24 further cancontinue to analyze the distribution of pixels or voxels within eachspatial region to provide updated dynamically varying imaging datareflecting changes in the distribution of pixels during suchinteractions or in between each of a sequence of interactions. As aresult, the tool function control 48 and tool motion control 46 candynamically adjust the process parameters associated with the automatedtool 26 for each spatial region containing an object of interest basedon the changing imaging data that has been provided by the imageanalysis 24.

In view of the foregoing it is to be understood and appreciated that theimage data 22 that is acquired by the imaging system 20 and the analyzedimage data from the image analysis 24 can be converted into processparameters to control interaction with one or more selected object ofinterest. For example, the control system 40, including the tool motioncontrol 46 and the tool function control 48, can be programmed tocompute processing parameters for an interaction protocol based on thedistribution of pixels or voxels for an object of interest or a featureof an object of interest determined from the image analysis 24 for thepixels or voxels. Other information, such as sensor data and thecategory of medium, can also be used in determining the interactionprotocol's processing parameters.

By way of further example, each object of interest, such as a cell or agroup of cells (e.g., cell colony) can include one or more morphologicalfeatures that can be readily determined by the imaging analysis 24according to the values of pixels or voxels for each object of interestand the distribution of pixels or voxels for the object of interest.That is, in addition to the distribution of pixels or voxels,corresponding morphological features corresponding to the values of therespective pixels or voxels can be combined for furthercharacterization. The morphological features can include, for example,cell morphology or morphology of a colony of cells, such as disclosed inthe above-incorporated U.S. Pat. No. 8,068,670.

The tool motion control 46 further can be programmed to control themotion of a tool relative to the stage or other substrate in which theobject of interest 14 is located. As mentioned above, the tool motioncontrol 46 can control motion and position of the tool inthree-dimensional space (e.g., via respective linear motors oractuators) based on the distribution of pixels or voxels for a givenobject of interest provided in the image data 22. Based on theestablished parameters for positioning the tool in contact with or at apredetermined distance separated from the object of interest, the toolfunction control 48 can in turn control the flow of the object ofinterest into an aspiration tool, again, based on the distribution ofpixels and based on the analysis of the values of pixels or voxels forthe given object of interest to be aspirated.

There can be any number of one or more aspiration schemes defined byrespective interaction protocols that can be stored in memory andselected for use in facilitating isolation and harvesting of cells. Theaspiration scheme may be selected among the options in the protocollibrary, such as based on the imaging features of the object of interest(e.g., size, density, shape, morphology, expression of selected surfacemarkers, or the like). For example, the tool function control 48 can beprogrammed to select an interaction protocol from the library ofinteraction protocols to implement an aspiration scheme based on theanalysis of the image data acquired for the objects located on the stageand in response to a user input, such as can be provided the userinterface 18. For example, the user input can be utilized to specify adesired type of interaction or type of aspiration as well as one or moreparameters associated with such operations. Additionally oralternatively, the user input can establish a minimum colony size anddefine a type and material properties of the media in which the objectsare being cultured. The selected aspiration scheme and the analysis ofthe image data can be stored into a knowledge base to further documentthe application of the aspiration scheme for harvesting the object ofinterest. The set of process parameters associated with each interactioncan also be stored in memory for subsequent analysis and evaluation. Inthis way, process parameters can be further adjusted for subsequentharvesting based on the evaluation of the effectiveness of the schemeand the parameters utilized to harvest the object of interest.

As a further example, one or more parameters, such as tip distance fromthe object, flow rates, pressures and positioning of the tool relativeto the object of interest can be evaluated relative to the results ofthe aspiration and harvesting process to determine the effectiveness ofsuch parameters during aspiration. Moreover, the user input can furtherset a type of interaction, such as sample, move, remove, or kill anobject of interest, depending upon whether it is determined that theobject of interest is to be retained, transferred or removed from themedium.

In some examples, a given colony or group of cells may require multipleinteractions or aspiration phases to displace unwanted cells or objectsand/or to retain desired cells in the medium or capture the desiredcells into the inner chamber of an aspiration tool for transfer.Accordingly, between each of the sequences of aspiration phases,calibration updates can be implemented dynamically to help optimize anext aspiration phase for the selected interaction protocol. Forexample, flow rates and distances can be adjusted, position of a needlecan be readjusted and different flow rates can be utilized based on thedistribution of pixels and voxels detected between each aspirationphase. As an example, the distribution of pixels before and/or after anaspiration sequence can be compared or correlated to ascertain processparameters for the next phase of aspiration.

The image analysis 24 further can be programmed to determine an objectprofile for each object of interest based on the analysis of thedistribution of pixels or voxels for the object of interest. As anexample, an object profile may be compiled from an integrated series ofalgorithm steps, each applying a specific operation or range of specificvariables including specific steps of imaging, image processing (e.g.,thresholds, filters, maps, masks, segmentation, correction) steps. Thesemay develop an object list that can be further refined by specificinclusion and exclusion testing criteria based on size, morphology,density, brightness, texture, gradients, proximity, shape, pattern andthe like. Individual (discrete) profiles for detection andclassification of specific object types may differ by one or more stepsor parameter range or threshold settings. The process parameters forcontrolling interaction such as aspiration with each object of interestthus can be selectively adjusted based on the object profile that hasbeen determined. As an example, the object profile can be assigned toeach discrete spatial region occupied by corresponding objects ofinterest and the object profile further can be updated dynamically basedon the distribution of pixels or voxels being acquired for each phase ofthe process.

After the objects of interest have been harvested into the inner passageof a tip (e.g., needle) of an aspiration tool, the stage motion control44 and tool function system 28 can cooperate to move the selectedobjects from the needle to a subsequent destination, which can reside onthe stage 12 or another location within the system 10. For example, ifthe objects or cells that have been aspirated into the needle fordeletion, the needle can be moved so that can be discarded to anappropriate disposal site. In other examples, the cells can betransferred to a new site, such as being re-plated into a medium foradditional growth and subsequent harvesting.

As mentioned above, the location of a tip of tool, which may be integralwith tool or be replaceable, is to be known in three directions, X-Y andZ. To enable accurate control for interactions with objects on thestage, these locations are calibrated to relate each motion system thatis involved in the interaction. Additional examples of calibration andinteraction between a tip of a tool and one more objects on a stage willbe better appreciated with respect to the example of the systemconfiguration depicted in FIGS. 2-10 . In the examples of FIGS. 2-10identical reference numbers refer to the same components and additionalreference can be made to the system 10 of FIG. 1 for integrated controlsand operations.

In the example of FIG. 2 , the system 100 includes a stage 110 that ismovable along at least two orthogonal axes, identified as X and Y. Thestage 110 can be movable relative to one or more tools 112 that caninclude a tip that extends axially from the tool to terminate in acorresponding distal end of such tip. The tool 112 can be controlled tomove the tip axially in a direction that is orthogonal to a planedefined by the X and Y axes of the stage, in the Z direction. The tipincludes a longitudinal central axis, identified at 114. In addition tomovement of the stage 110 along the X and Y axes with respect to asystem frame or housing, schematically indicated at 115, the tool 112can also move along the Y axis via operation of the corresponding motorsto move tool holder arm 116 with respect to mounting arm 118. The toolcan also be moveable in the Z direction via control of a linearactuator. For example, the Z-axis motor of the tool 112 can beimplemented as mechanical actuator (e.g., a ball-screw servo motor, alead-screw motor, etc.), a linear motor, a segmented spindle motor, amoving coil motor (e.g., rotary, linear or rotary-linear), or the like.

As a further example, the tool holder arm 116 can be movable in theY-direction via actuation of a corresponding linear motor or otheractuator. The tool 112 can also be moveable in the Z-direction withrespect to the stage 112 via actuation of the corresponding motor (e.g.,a linear actuator). Similarly, the stage 110 can be movable in the Xdirection in response to actuation of the corresponding linear motorthat is attached the stage 110 and corresponding base (e.g., frame orhousing) of the system 100, and in the Y direction in response to anactuation of another linear motor fixed to the stage 110 and to the baseof the system 100. The stage 110 includes a surface 122 on which variouscomponents can be located.

As one example, the tool 112 can be implemented to include a hollow bodythat is fluidly connected to a source of fluid which can be activated toadd or remove fluid through the hollow body and a corresponding tip thatis attached and extends from to the hollow body. For instance, a tip canbe removably attached to the hollow body, as mentioned above such as bya friction fitting between an interior sidewall of the tip and the outersidewall of the mandrel of the tool body. Other types of attachments,such as threaded fasteners and fittings, may also be utilized to securethe tip with respect to the tool 112. Since, in some examples, the tipis removable with respect to the tool, each tip that is attached to thetool can result in spatial variation of the distal end of the tip inthree-dimensional space, including in X and Y coordinates of the stage110 as well as in a Z direction that is orthogonal to the surface of thestage. Accordingly, the system 100 employs calibration to resolve theposition of the tip spatially.

In example of FIG. 2 , two well plates 124 and 126 are positioned spacedapart from each other on the surface 122 of the stage. While the exampleof FIG. 2 demonstrates two well plates 124 and 126 it is understood thatthe stage can accommodate any number of one or more than one platesaccording to application requirements. Each of the plates 124 and 126can include one or more wells into which cells can be received andremoved via the one or more tools (e.g., syringe tool apparatuses) 112implemented within the system 100. As disclosed herein, the tools (e.g.,a cutting tool, a scraping tool, a stamping tool or a stirring tool) canalso perform other types of interactions with the cells or other objectsor media that may be disposed in the wells 128 and 130 on each of therespective plates 124 and 126. Additionally, the enclosure 115 isdesigned to maintain environmental conditions within the enclosure tofacilitate growth and cleanliness of cells and other objects that may beplaced onto the well plates 124 and 126.

In some examples, the system 100 can also include a tip sensing system140, which may be integrated into the stage 110 and configure tointeract with the tip of one or more tools. The tip sensing system 140can include hardware that includes sensors 142 and 144 for detecting aposition of a tip of the tool 112 in each of the X and Y axes of thestage 110. The sensor 142 can be configured to detect the Y position ofthe tip of the tool 112 and the sensor 144 can be configured to detectthe X position of the tip of the tool.

As a further example, the tip sensor 140 can be a non-contact sensor,such as using optical interrupter devices oriented 90 degrees from eachother in the direction of the X- and Y axis. When the tip sensor 140 isutilized to detect the X and Y positions of the tip, the tool 112 can bepositioned at a predefined X and Y position with respect to the arms 116and 118 relative to the housing 115 of the system 100. This predefinedlocation (e.g., referred to herein as the tip reference position) can beidentified and stored in memory as to be reproducible for eachsubsequent tip that may be attached to the tool 112. As another example,the tip sensor 140 could be implemented using an X-Y laser gauge fixedwith respect to the stage, which outputs not only the position of thetip with respect to the stage coordinates, but also the diameter of thetip.

As disclosed herein, the imaging system can include a camera thatprovides an object field of view 146 at a corresponding location thatcan remain fixed with respect to the housing 115 of the system 100. Thefield of view 146 can include an optical axis 148, demonstrated at thecenter of the field of view 146. Thus, in order to capture an image of adesired location on the surface of the stage 110, stage motion system(motion system 16 of FIG. 1 ) can selectively adjust the position of thestage 110 in the X and/or Y axes so that the field of view 146 containsthe desired object or other target of interest such as located in acorresponding well 128 or 130. The size and resolution associated withthe field of view 146 can be selected by configuring optics 58 of thecorresponding imaging system 20, as disclosed herein, automatically orin response to a user input.

FIG. 3 depicts an example where the stage 110 has been moved to a firstposition in which the tip 114 can be moved into contact with a locationat the surface of the stage such as in a corresponding well. Theparticular location at which a reference tip contacts the stage can bearbitrary or predefined. The tool 112 can be moved along the Z axis 114in the direction of the stage 110 until the tip contacts the stage or anobject located on the stage such as the well or medium residing in thewell. In response to engagement between the tip and contact site on thestage, a fiducial marker can be transferred onto the contact site. Themarker can include ink or any other marker material that can be visuallydifferentiated from the surroundings on the surface as to be detected bythe imaging system. For example, the ink can be applied to the distalend of the tip prior to moving the tip into contact with the stage 110.

By way of example, the ink can be applied to the end of the referencetip. The distal end of the tip can be moved to contact an empty sampletray at a known empty location. For instance, the tool motion system cancontrol the tip to contact the tray at a plurality of different X-Ylocation by moving the tip along its Z axis to stamp the tray with theink that is on the end of the tip, thereby creating a fiducial mark(e.g., a circle) on the sample tray. After the fiducial mark (or aplurality of markers) has been transferred to the surface of the stage110, stage motion system is activated to adjust the position of thestage 110 so that the fiducial marker is within the field of view 146 ofthe imaging device (e.g., microscope), such as demonstrated in FIG. 4 .

In some examples, the stage may be moved in response to user inputs(e.g., in the X and Y directions to position the fiducial marker withinthe optical field of view 146). In other examples, an automated methodcan be employed to adjust the position of the stage 110 with respect tothe object field of view into the fiducial marker resides within theobjects field of view. As yet another example, a combination of userinputs and automated detection can be utilized to adjust the positioningof the stage with respect to the optical field of view 146. For example,the stage position can be adjusted to place the fiducial marker alongwith the optical axis 148. Once the fiduciary marker is located at thedesired position within the object field of view 146, as shown in FIG. 4, the coordinates of the stage 110 can be stored in memory.

The system calibration function (e.g., function 54) of the controlsystem can in turn calculate a difference between coordinates of thestage (e.g., X-Y coordinates) when the fiduciary marker was applied bythe tip of the tool and the coordinates of the stage when the image wascaptured that contains the representation of the fiduciary marker withinthe objects field of view 146. The difference between the spatialcoordinates of the stage at the respective positions for each fiducialmarker determines an optical offset between the tip position(corresponding to the reference tip position) and the optical access.The positions of the stage can be determined from outputs of X and Ylinear encoders that are associated with the respective linear actuatorsutilized to move the stage along each of its X and Y axes.

By way of example, the fiduciary marker can correspond to an ink dothaving a diameter that is approximately 700 micrometers or less and thecoordinates of such fiduciary marker can be determined according to apixel or to a centroid of a group of pixels that contain the fiduciarymarker from the image that was captured from the field of viewcontaining the fiduciary marker. If the centroid or individual pixel forthe fiduciary marker is not aligned with the optical axis, the imagemetadata (metadata 23 of FIG. 1 ) that specifies the resolution of theimage can be employed to ascertain its spatial position of the pixel orcentroid of pixels.

The spatial offset that is determined can be utilized to reproducibly tomove the tip relative to a desired target that is identified within afield of view of a given image, such as to perform a desired interactionat the target object location. As disclosed herein, examples of someinteractions that can be performed with respect to objects on the wellplates 124 or 126 can include cutting material on the tray, scrapingmaterial on the tray, stamping material on the tray, stirring a mediumon the tray or aspirating (e.g., picking) objects from the tray and/ortransferring objects to one or more destinations.

For examples where the tip is a removable tip, such that it can beremoved from the tool and replaced with another tip, the control systemcan perform additional calibration to ascertain a tip-to-tip spatialoffset for each new tip that is used. In the following examples, it ispresumed that the set of images stored as image data correspond to thesame sets of plates 124 and 126 that are positioned at the samelocations on the stage 110. In this way the calibration and anyinteractions are performed with the respect to a common set of images.

As shown in FIG. 5 , for example, a new tip can be placed onto the tool(e.g., mandrel) from tip storage 1202, such as after the previous tip(e.g., the reference tip or another tip) has been removed from the tool,such as disclosed herein. Once the new tip has been placed on the bodyof the tool, such as by automatic placement from the tip storage 120,the tip sensor 140 is activated to ascertain the location of the tip inX and Y coordinates for the stage. Such tip coordinates are determinedfor each tip used, including the reference tip. The tip-to-tip offsetcan be calculated as the difference between the tip coordinates of thereference tip and each other tip.

For example, as shown in FIG. 6 , the tool 112 can be repositioned backat the reference tip position, and the stage 110 can be moved in the Xdirection so that the Y coordinate for the tip of the stage can beidentified by the sensor 142, as indicated by axis 152. Similarly, asshown in FIG. 6 , the stage can be moved in the Y direction so that an Xcoordinate for the distal end of the tip is identified by the sensor144, as demonstrated by axis 154. The intersection between the axes 152and 154, as determined from sensors 142 and 144, corresponds to the tipposition (X, Y coordinates) for a given tip. Based upon the tipcoordinates for the reference tip and each other tip, a correspondingtip offset can be determined as the difference between the tipcoordinates of the reference tip and each respective other tip.

The corresponding tip offset for a given tip can in turn be applied tothe spatial offset determined for the reference tip (see, e.g., FIGS. 3and 4 ) to provide the new optical offset. For instance, the X opticaloffset can be computed as the sum of the X coordinate for the opticaloffset and the X value of the tip offset, and the Y optical offset forsuch given tip can be the sum of the Y coordinate for the referenceoptical offset and the tip offset in the Y direction. The new opticaloffset is utilized by the control system 40 for controlling the stagemotion system for positioning the distal end of the new given tip at adesired target location identified in a corresponding captured image. Asa result interactions between the tip and objects on the stage can beimplemented with a high degree of accuracy and reproducibility. Forexample, the precision of the X-Y tip sensor 140 is can be about +/−5microns or better.

FIG. 8 demonstrates a further example of identifying the tip locationwith respect to one of the X and Y axes. In the example of FIG. 8 , thetip sensor 150 includes an optical transmitter 152 and optical receiver154. The optical transmitter can transmit a beam of light (e.g., laser)155 above the surface 122 of the stage 110 that is detected at thereceiver 154. As the tip is spaced in the Z direction a predetermineddistance above the stage 110 on which the sensor 150 is fixed, the stagecan be moved in a direction orthogonal to the beam (along one of the Xor Y axes) so that a distal end 156 of the tip 158 disrupts the beam.The disruption of the beam generates a tip sensor signal 160 to triggerrecording respective coordinates for each beam for the axis that extendsorthogonal to the beam. The coordinates of each beam may be known withrespect to the X and Y coordinates of the stage or the beam coordinatesmay be relative to predetermined reference coordinates of the stage.Either way, the position information provided by each of the sensors 142and 144 can be utilized to provide the position of each tip, which canbe utilized to ascertain a precise tip offset between any two tips basedon stage coordinates recorded when each beam is interrupted during thetip sensing phase.

Additionally, since the distance 162 can be determined in the absence ofthe distal end of the tip contacting the stage 110 or another structuredisposed on the stage, there is a risk of contamination 156 through suchlocation process. FIG. 8 also demonstrates a block diagram that can beutilized to determine the tip-to-tip offset as well as the correspondingspatial offset between the tip location and the optical access of theimaging system.

As a further example, a tip sensor processor 170 receives the tip sensorsignal 160 to ascertain corresponding tip position along a given one ofthe X or Y axis. It is understood that each of the sensors 142 and 144can implement similar processing to ascertain the tip location along theX and Y axes. In some examples, the tip sensor processing can alsodetermine the distance 162 that can be combined with the known locationof the tip sensor with respect to the X and Y coordinates of the stage110 to provide absolute position data 172 for the X and Y positiondepending on which sensor 142 or 144 has detected the positioning of thetip 158. The tip position data 172 can store tip position for one ormore tips that are utilized, such as including the reference tipposition and the position of each subsequent tip that is being sensed bythe system 150.

A tip-to-tip offset calculator 174 thus can compute the tip-to-tipoffset based on the tip position data for the reference tip and anothertip. The tip-to-tip offset can in turn be provided to a spatial offsetupdate function 176. The spatial offset update can aggregate thereference spatial offset that is stored in memory with the tip-to-tipoffset that has been calculated to in turn provide the updated spatialoffset data that can be stored in memory and in turn utilize to positionthe tip 158 with respect to a desired target object that has beenidentified in the field of view of a captured image.

In addition to calibrating the position of the tip with respect to X andY coordinates of the stage 110, systems and methods disclosed herein canbe implemented to calibrate the tip position with respect to the Z axis,which is orthogonal to the X and Y axes of the stage. The Z axiscalibration thus can relate a height of the distal end of the tip withrespect to the surface of the stage to further facilitate controllinginteraction with one or more objects that may reside in a mediumpositioned on a surface of the stage. For example, plates that may bepositioned on the surface of the stage may have non-uniform thicknessesas well as other topographical variations across the surface thereofthat may need to be accounted for. Additionally or alternatively,different tips that can be positioned onto the tool, such as disclosedherein, for picking and/or for placing objects with respect to thestage. When attached to the tool, each of the different tips can providespatial variations in the Z direction as well as the X and Y directions,even when the tips are of the same size and design. These and othervariations, if not accounted for, can result in errors when interactingwith objects at various locations on the stage.

FIGS. 9 and 10 depict an example of an approach that can be utilized todetermine height of a tip 158 with respect to a local region of thestage or other object that may be positioned on a surface of the stage122. As mentioned the surface of the stage may or may not be planarsurface. The calibration along the Z axis can be utilized to reconcilesmall (e.g., micron level differences) between a plane of the tip and asurface plane of the stage to enable precise control of the height ofthe tip with respect to the surface of the stage. In the example of FIG.9 , the tip 158 is positioned a distance “h” above the surface 122 ofthe stage or an object positioned on the stage. A sensor 180 isimplemented to detect force applied to the tip in the Z direction (e.g.,along the Z axis of the tip 158).

By way of example, the force sensor 180 can be implemented as part ofthe tool, such as integrated into the tip holder, via mount system toinsure adequate sensitivity along the Z axis. In other examples, theforce sensor 180 might be a strain gauge that is part of the tool orintegrated into the tip itself. As another example, the sensor could bepositioned on the stage or be implemented partially on the tip or tooland partially on the stage to provide information to determine Z axisheight in reference to the bottom of the sample plate. The force sensor180 operates as a highly sensitive scale measuring the weight of thesmart syringe, such that any pressure applied either up or down providesa corresponding output indicative of the sensed force. The force sensorsignal can be provided to the control system via a communications link.For example, the communications link may be a wireless link (e.g.,Bluetooth or other short range wireless communications technology) or aphysical link (e.g., optical or electrically conductive cable). Theforce sensor output can be stored in memory as force sensor output data182 for processing to ascertain the z-axis position of the tip 158.

The control system 40 is programmed to determine the height of the tipbased on the force sensor data and Z-position data 184 obtained during aZ axis calibration mode. For example, tip 158 begins at a start positionwhere Z-axis position data 184 is recorded based on an output of aZ-axis encoder associated with the tool. The control system 40 providestool motion control instructions to move the tip in the Z directiontoward the surface 122. The distal end of the tip traverses a distancefrom its start position up to a distance corresponding to the height ofthe tip from the stage, producing corresponding Z-position data duringsuch movement. In response to the distal end of the tip 158 contactingthe surface 122, the force sensor 180 can provide a corresponding signalrepresenting the sensed force, which indicates contact between the tipand the solid surface. A height calculator 186 (e.g., of the controlsystem 40) computes the distance of travel, corresponding to tip heightdata 188, based on a difference between the Z position data 184 at thestart position (FIG. 9 ) and the Z position data when contact occurs(FIG. 10 ). For example, the contact can be ascertained from the forcedata 182 indicating sufficient force to indicate a solid object, such asthe surface 122, and not a fluid medium disposed thereon. That is, thecontrol system 40 can discriminate between contact with a solid surfaceand a fluid medium.

In addition to the tip height data specifying the distance 178 betweenthe tip and the surface 122, the tip height data can also specify the Xand Y coordinates (e.g., stage coordinates) where the tip height ismeasured (e.g., a tip height measurement location). The tip height data,including the height in the Z direction and associated X, Y coordinatesfor such location, can be stored in memory for a plurality of X, Ycoordinates across the surface of the stage. The tip height data 188 canbe utilized to provide a corresponding topographical map of the surfaceof the stage 122 or other object disposed thereon for one or moreportions of the surface on the stage, such as shown in FIG. 11 . Forexample, the tip height can be utilized to create a topographicalsurface map based on measuring height for one or more locations acrossthe surface of the stage, which tip height measurements and associatedmeasurement locations (e.g., in stage X,Y coordinates) can be stored inmemory.

FIG. 11 demonstrates an example of a topographical map with a surface,such as an unoccupied surface of a well plate that is positioned on thestage 110. It is understood that such a detailed map is not required forthe entire surface sense the tip plate is only relevant or needed forlocations adjacent to each target site.

The control system 40 can employ the stored tip height measurements tocontrol the tip height at or near one or more desired locations on thesurface 122 such as for interacting with objects that have beenidentified in the corresponding image. For example, prior to eachinteraction at a specified X,Y coordinate (e.g., determined from imageanalysis 24 as disclosed herein), the Z axis tip height calculator 186of calibration function 54 can be utilized to determine the height ofthe tip (e.g., corresponding to the distance of the Z-direction betweenthe tip and the surface 122) at a location that is adjacent to butspaced apart from the target site. Similarly, the corresponding tipheight ascertained at the adjacent location or a series of adjacentlocations can be utilized to specify the tip height as to control theinteraction with objects at each respective target site. If the tipheight is already known at a measurement location that is within apredetermined distance of a target site, the known tip height can beutilized to control the interaction at each target site that is withinsuch predetermined distance. By determining tip height at one or moremeasurement locations adjacent to a target site, the height of the tipcan be controlled precisely during interaction at the target sitewithout having to potentially damage the object at the target site.

In view of the foregoing structural and functional features describedabove, methods that can be implemented will be better appreciated withreference to FIGS. 12-14 . While, for purposes of simplicity ofexplanation, the methods of FIGS. 12-14 are shown and described asexecuting serially, it is to be understood and appreciated that suchmethods may not be limited by the illustrated order, as some aspectscould occur in different orders and/or concurrently with other aspectsfrom that shown and described herein. Moreover, not all illustratedfeatures may be required to implement a method. The methods or portionsthereof can be implemented as instructions stored in a non-transitorystorage medium as well as be executed by a processor of a computerdevice, for example. Additionally, in some implementation, two or moreof the methods of FIGS. 12-14 (or portions of two or more such methods)may be combined together as part of a workflow for calibrating and/orimplementing an interaction protocol.

FIG. 12 depicts an example of a calibration method 200 to facilitatepositioning a stage to provide for interaction between a tip and one ormore objects identified in an image. The method 200 begins at 202 inwhich a marker is provided at a touchoff location. For example, acorresponding tool can be moved to a predefined reference position andmoved in the Z-direction to contact the surface of the stage and place amarker at such touchoff location in response to such contact. The markercan be ink or any other substance that may be visible via the imagingsystem (e.g., system 20 of FIG. 1 ). At 204, a reference location forthe tip can be identified. For example, the reference location for thetip can corresponding to the coordinates of the movable stage (e.g.,stage 12 or 110) in two or more dimensions such as according topositions provided by encoders for the X,Y directions for linearactuators that are employed for adjusting the position of the stage.

At 206, the marker is moved into the field of view of the imagingdevice. For example, the stage position can be adjusted to align themarker within a field of view from the imaging system (e.g., camera)that is positioned orthogonally and spaced apart from the surface of thestage onto which the marker is provided at 202. The movement of themarker to a prescribed position within the field of view can beimplemented manually (in response to user inputs), automatically or canemploy a combination of manual and automatic motion controls.

At 208, the location of the marker within the field of view can beidentified. For example, the location of the marker within the field ofview can be identified based on the coordinates (e.g., X,Y position) ofthe stage after the marker has been moved into the field of view at 206.The identified location can itself be the X,Y position of the stage orit can be the X,Y position of the stage in combination with distancebetween one or more pixels that represent the marker in an imagecaptured by the image device relative to an optical reference in theimage.

For example, the optical reference in the image can correspond to acenter of the field of view for the captured image or to one or morepixels at the periphery at such captured images. For instance, if themarker is moved into alignment at the Z-direction with the referencepixel or pixels in the field of view, then the identified location at208 can represent the position of the stage. Otherwise the pixeldistance can be applied to the position of the stage to represent thelocation of the marker within the field of view.

At 210, the spatial offset between reference location for the tip andthe field of view is determined. The spatial offset can be determined,for example, based on a difference between the identified location forthe tip at 204 and the identified location for the marker at 208. Thespatial offset for the tip and the field of view can be stored in memoryas an optical offset that is utilized at 212 to position the stagerelative to the tip for any number of one or more subsequentinteractions between the tip and objects on the stage. For example, thecontrol system can apply the spatial offset to move the stage so thatany object identified in any captured image can be aligned with respectto the tip to enable interactions between the tip and such identifiedobjects. As disclosed herein, the interactions can include one or moreof cutting material from the stage (e.g., from a well plate), scrapingmaterial, stamping material on the stage, stirring or agitating a mediumon the tray or aspirating (e.g., picking) objects from the tray and/ortransferring objects to one or more destinations on the stage orelsewhere.

FIG. 13 depicts an example of a method 250 to control interaction withobjects. The method 250 begins at 252 with capturing one or more imagesvia the imaging device (e.g., imaging system 20). The captured image canbe for a field of view on the surface of the stage (e.g., stage 12 or110) such as including various objects, including biological objects orthe like that may be disposed within well plates as is known in the art.For each image that is captured at 252 metadata is stored with the imageat 254 (e.g., metadata 23 embedded in or otherwise associated with imagedata 22). The metadata includes position of the stage at the time ofeach image as well as the indication of image resolution for eachcaptured image. The image resolution and stage position thus can beutilized to ascertain an absolute position in the X,Y coordinates of thestage for each pixel in each captured image. For instance, one or morereference pixels in a given image (e.g., a center location and/or alocation on a periphery image) can be used as reference pixels that havea known position with respect to the coordinates of the stage. Since theresolution is known, each pixel can be assigned a dimension such thatthe number of pixels between the reference position can be utilized toascertain a relative or absolute position with respect to the X,Ycoordinates of the stage. While the foregoing has been described astwo-dimensional image, similar metadata can be provided for athree-dimensional image.

At 256, the captured image can be analyzed to determine one or moretarget objects in the image. For example, the image analysis at 256 canbe implemented according to the above-incorporated U.S. Pat. No.8,068,670. Those skilled in the art may understand and appreciate otherimage analysis techniques that may be utilized to detect target objectson the stage. At 258, the location of the target object is determinedbased on the stored metadata for the captured image that was analyzed ofone or more objects.

At 260, the stage can be moved relative to the tool to align thelocation of the target object or objects with the tip of the tool. Forexample, the control system (e.g., stage motion control 44) can controlthe motion of the stage along each of the X,Y positions to align the tipposition with the target object. If the target object is larger than thetip of the tool, and the interaction is to occur over a spatial areathat involves multiple sequential interactions, a correspondinginteraction protocol, such as disclosed herein can be utilized tocontrol the stage to move to a sequence of locations for tip alignmentand corresponding interaction at each of the sequential locationsdistributed across the surface area of the identified object.

At 262, the tip can be controlled to interact with the object such as bycontrolling motion of the tip along the Z axis to a location that isspaced apart from the contact surface of the stage or plate that may bepositioned thereon and the corresponding interaction may then beactivated according to the selected interaction protocol. As oneexample, the hollow tip can be inserted into a medium and aspirated anddraw in a volume of media and corresponding biological objects growingtherein. This may be repeated at the plurality of sequential locationsas disclosed herein. Other forms of interaction disclosed herein can beperformed at the identified locations disclosed herein.

From 262, the method may return to 256 to further analyze one or more ofthe images to determine additional target objects for which interactionmay be desired and the method may proceed accordingly. Alternatively,the method can proceed from 262 to 260 to in turn move the stage to asubsequent sequential location to align the tip of the tool with anotherpart of the target object to enable the interaction at 262 for each ofthe subsequent sequential locations that have been specified. For eachpositioning of the stage with respect to objects in an image, themetadata of the image is utilized for such positioning. The method canrepeat until the procedure has been completed according to the selectedinteraction protocol(s).

FIG. 14 depicts an example of another method 300 that can be utilized tofacilitate positioning a tip with respect to the stage along the Z axis.The method 300 can be considered a Z axis calibration method, forexample. The method 300 begins at 302 in which a target location isdetermined. The target location can be determined by specifyingcoordinates, such as may be determined from image analysis (e.g.,analysis 24) performed for one or more captured image or may otherwisebe identified. The target location for example, may include biologicalor other objects of interest, such as art to be picked from a mediaresiding on the stage. In some examples, target objects can beconsidered objects that are to be destroyed and removed from the stageto allow further growth and culture of desired biological objects.

At 304, the stage is moved relative to the tool to position the tipadjacent to but spaced from the target location determined at 302. Theadjacent location can be free from the desired target object, such as insituations where it is desired not to contact the objects duringcalibration. For example, where a colony of cells have been identifiedon the stage within a given surface location, an adjacent portion of thestage can be defined as the adjacent area (e.g., positioned greater than10 microns, such as about 50-100 microns from the identified targetlocation) to which the tip can be aligned by movement of the stage at304.

At 306, the tip can be controlled to contact the stage at such adjacentlocation. For example, a corresponding linear actuator can be controlledto move the tool and the tip that is attached thereto in the directionof the stage until contact between the tip and stage is detected (e.g.,by Z-axis force sensor 180). At 308, the difference between the startingposition and the tip and the distance traveled in the Z-direction (e.g.,as measured by a Z-axis encoder) can be employed to determine the heightof the tip at the adjacent measurement location. Since the adjacentmeasurement location is sufficiently close although spaced from thetarget object location, it is assumed that the variation in the surfaceof the stage or other contained object disposed thereon will berelatively small. A threshold for the adjacent measurement location canbe preset or user defined according to application requirements, such asin response to a user input. Thus, after Z-axis calibration for tipheight at the adjacent measurement location, at 310, the stage can bemoved relative to the tool to position the tip aligned axially with thetarget location.

At 312, the tip height determined at 308 can be utilized to control thetip to interact with the object and/or medium at the target location ina desired manner according to a selected interaction protocol. Forexample, since the height of the tip is known at the adjacentmeasurement location, the interaction protocol can use that height toensure that the tip is not moved in the Z-direction, unless that isrequired by the interaction protocol. The method 300 can return from 312to 302 to repeat 304-312 for each subsequent target location as part ofthe interaction protocol. In some cases, where the next target locationare considered sufficiently close to the adjacent measurement locationwhere the height was determined at 308, from 312 the method can insteadreturn to 310 to remove the tip of the tool to the next target locationand employ the same tip height over a plurality of interaction.

FIG. 15 depicts an example of captured images 402, 404, 406 and 408demonstrating different steps of an interaction protocol associated withmonolayer picking. For example, the captured image 402 corresponds to asource location in which a plurality of objects of interest have beenidentified such as corresponding to a colony of area. For example, theobject of interest can be identified by image analysis (e.g., analysis24) such as can include the colony area, colony density, cell number,cell marker area expression or a combination thereof. The captured image404 demonstrates the source location for the object of interestfollowing biopsy of the cells, such as using a tip and aspirating fromthe source location containing the desired target objects. In image 404,inner and outer diameters of the harvest tool are demonstrated asconcentric rings 410 and 412, respectively. The captured image 406demonstrates the cells after being transferred to a different locationfor which the captured image 406 was obtained for a corresponding fieldof view. The cells for example demonstrate growth for the transferredcells for a period of about 24 hours. The image 408 demonstrates thetransferred cells at a later time period (e.g., about six days).

FIGS. 16A and 16B depict a schematic illustration of first and secondaspirations performed for an object, demonstrated at 450 and 452. Thesyringe motion and/or stage motion systems can cooperate to position atip 458 of a hollow needle for each aspiration. For the first aspiration450, for example, where an object (e.g., a colony) 454 is disposed in atwo-dimensional medium, an initial centroid 456 for the object can becomputed (e.g., by image analysis 24). Corresponding process parametersfor a selected interaction protocol can be applied based on the relevantdata (e.g., image data, sensor data, syringe motion data, aspirationdata) to control the first aspiration that is performed by positioningthe tip 458 into the medium at the centroid 456 spaced a distance.

Following the first aspiration (FIG. 16A), the colony geometry can shiftsuch as demonstrated schematically at 452 in FIG. 16B. Prior toperforming the second aspiration, process parameters can be recomputedand stored at least in part as updated aspiration data for the nextaspiration. For instance, this can include computing (e.g., by the imageanalysis 24) a new centroid 456′ of the remaining target object based onthe updated image data that is captured by moving the target object 454into the field of view of the image capture device (e.g., camera 60).The tool function control of the control system 40 can employ theupdated process parameters to perform the second aspiration according tothe selected protocol. This process can be repeated until harvesting hasbeen completed for the target object.

As another example, FIG. 17 demonstrates another approach that can beutilized for harvesting cells from a colony according to a selectedinteraction protocol. For instance, an original colony object can beidentified based on the image analysis, as demonstrated at 500. Based onthe image analysis, the aspiration tool and fluidic controls can computean initial (e.g., original) harvest strategy 502. The harvest strategy502 can include a plurality of target sites 504 at which aspiration isto be performed sequentially as well other process parameters forcontrolling aspiration at each of the sites. Following theinitial/original harvesting at the sites 504, the colony object can bemoved to within the field of view of the imaging system to acquire imagedata (e.g., one or more images) representing the current state of thecolony object, shown at 506. The corresponding image data can beutilized to compute a second, updated harvest strategy 508 to facilitateharvesting remaining cells in the colony object at new target sites,such as shown at 510. While in this example, the second harvest strategyis computed after completion of the first strategy, it will beappreciated that the harvest strategy can be variable and dynamicallyupdated based on image data acquired during any number of sequences ofaspiration specified by the interaction protocol.

In view of the foregoing structural and functional description, thoseskilled in the art will appreciate that portions of the invention may beembodied as a method, data processing system, or computer programproduct. Accordingly, these portions of the present invention may takethe form of an entirely hardware embodiment, an entirely softwareembodiment, or an embodiment combining software and hardware.Furthermore, portions of the invention may be a computer program producton a computer-usable storage medium having computer readable programcode on the medium. Any suitable computer-readable medium may beutilized including, but not limited to, static and dynamic storagedevices, hard disks, optical storage devices, and magnetic storagedevices.

Certain embodiments of the invention have also been described hereinwith reference to block illustrations of methods, systems, and computerprogram products. It will be understood that blocks of theillustrations, and combinations of blocks in the illustrations, can beimplemented by computer-executable instructions. Thesecomputer-executable instructions may be provided to one or moreprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus (or a combination ofdevices and circuits) to produce a machine, such that the instructions,which execute via the processor, implement the functions specified inthe block or blocks.

These computer-executable instructions may also be stored incomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory result in an article of manufacture including instructions whichimplement the function specified in the flowchart block or blocks. Thecomputer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

What have been described above are examples. It is, of course, notpossible to describe every conceivable combination of components ormethods, but one of ordinary skill in the art will recognize that manyfurther combinations and permutations are possible. Accordingly, theinvention is intended to embrace all such alterations, modifications,and variations that fall within the scope of this application, includingthe appended claims. Where the disclosure or claims recite “a,” “an,” “afirst,” or “another” element, or the equivalent thereof, it should beinterpreted to include one or more than one such element, neitherrequiring nor excluding two or more such elements. As used herein, theterm “includes” means includes but not limited to, the term “including”means including but not limited to. The term “based on” means based atleast in part on.

What is claimed is:
 1. A system comprising: a stage; a stage motionsystem adapted to move the stage along orthogonal axes of the stage; atool holder supporting a tool above a surface of the stage, wherein thetool includes a body extending toward the stage and terminating in adistal end thereof, wherein the distal end of the tool body isconfigured to support a given tip of a plurality of tips, and each ofthe plurality of tips is replaceable on the distal end of the tool body;a tool motion system configured to position the tool relative to thestage; a tip position sensor fixed with respect to the surface of thestage and configured to provide first tip position data representing atip location of the given tip of the plurality of tips with respect toat least one orthogonal axis of the stage when the tool is positioned ata reference position, wherein in response to replacing the given tip ofthe plurality of tips with an other tip of the plurality of tips, theother tip has a tip location different than the given tip with respectto the at least one orthogonal axis of the stage; and a control systemconfigured to: control the tool motion system to position the tool atthe reference position while the tool body supports the other tip;control the tip position sensor to detect a spatial location of theother tip and provide second tip position data representing the tiplocation of the other tip when the tool is positioned at the referenceposition; determine a tip-to-tip offset between the tip location of thegiven tip and the tip location of the other tip based on the first andsecond tip position data; aggregate the tip-to-tip offset with apredetermined spatial offset to provide an aggregated spatial offsetrepresentative of a spatial offset between the other tip and an opticallocation within a field of view (FOV) of an imaging device having aknown spatial position relative to the stage, the predetermined spatialoffset being representative of a spatial offset between the given tipand the optical location; and control the stage motion system toposition the stage relative to the other tip based on the aggregatedspatial offset to enable the other tip to interact with a targetlocation on the stage identified in one or more images provided by theimaging device.
 2. The system of claim 1, wherein the tip positionsensor further comprises: an optical transmitter configured to transmita beam of light; and an optical receiver configured to receive the beamof light, the given tip being spaced apart from the surface of the stageat a predetermined distance, the stage being configured to move in theat least one orthogonal axis of the stage orthogonal to the beam, suchthat a distal end of the given tip disrupts the beam to generate thefirst tip position data, wherein in response to replacing the given tipwith the other tip, the other tip being spaced apart from the surface ofthe stage at the predetermined distance, the stage being configured tomove in the at least one orthogonal axis of the stage orthogonal to thebeam, such that a distal end of the other tip disrupts the beam togenerate the second tip position data.
 3. The system of claim 1, whereinthe control system is configured to: control the stage motion system toposition the stage at a plurality of different positions along theorthogonal axes of the stage; receive stage position data representingthe position of the stage in the orthogonal axes of the stage identifyrespective spatial locations based on the stage position data; anddetermine the predetermined spatial offset based on the respectivespatial locations.
 4. The system of claim 3, wherein the imaging deviceis associated with the stage and spaced apart from the tool, wherein thestage motion system is configured to: position the stage at a firstposition of the plurality of different positions to position a visiblemark on the surface of the stage outside the FOV of the imaging device;and position the stage at a second position of the plurality ofdifferent positions to position the visible mark in the FOV of theimaging device.
 5. The system of claim 4, wherein the control system isconfigured to: identify first spatial coordinates for the visible markin the orthogonal axes of the stage in response to the stage beingpositioned at the first position; cause the imaging device to capture atleast one image for the FOV in response to positioning the visible markwithin the FOV of the imaging device; identify the optical locationassociated with at least one pixel in the FOV based on the at least oneimage, wherein the optical location is associated with respectivespatial coordinates in the orthogonal axes of the stage corresponding tosecond spatial coordinates; and identify the second spatial coordinatesfor the visible mark with respect to the orthogonal axes of the stagebased on the optical location in response to the stage being positionedat the second position, wherein the predetermined spatial offset isdetermined based on the first and second spatial coordinates.
 6. Thesystem of claim 5, wherein the other tip of the tool comprises a marker,wherein the control system is configured to position the other tip ofthe tool relative to the stage to contact the surface of the stage totransfer the marker to provide the visible mark on the surface of thestage in response to the stage being positioned at the first position,the first spatial coordinates being identified based on the stageposition data in response to the stage being positioned at the firstposition, and the second spatial coordinates being identified based onthe stage position data in response to the stage being positioned at thesecond position.
 7. The system of claim 6, wherein the control system isconfigured to position the stage relative to the other tip further basedon a tip height for the other tip to enable the other tip to interactwith the target location on the stage.
 8. The system of claim 7, whereinthe tool motion system is configured to move the tool relative to thestage along an axis that is orthogonal to the surface of the stage toenable the other tip to contact the surface of the stage or an object atthe target location on the surface of the stage, the system furthercomprising a contact sensor to detect the contact between the other tipand the stage or the object on the surface of the stage.
 9. The systemof claim 8, wherein, prior to interaction with the target location onthe surface of the stage, the control system is configured to: identifyfirst tip spatial coordinates for the other tip of the tool in the atleast one orthogonal axis of the stage in response to the tool beingpositioned at a given location relative to the stage; cause the toolmotion system to move the other tip of the tool along the axis that isorthogonal to the surface of the stage to contact the surface of thestage at the given location that is adjacent to but spaced from thetarget location on the stage, wherein the contact sensor is to generatea corresponding signal indicative of contact between the other tip ofthe tool at the surface of the stage at the given location; identifysecond tip spatial coordinates for the other tip of the tool in the atleast one orthogonal axis of the stage based on the correspondingsignal; and determine the tip height of the other tip based on the firstand second tip spatial coordinates, the control system being configuredto control the interaction of the other tip with the target location onthe surface of the stage further based on the tip height.
 10. The systemof claim 9, wherein the control system is configured to cause the othertip of the tool to interact with one or more objects of interest at thetarget location based on the aggregated spatial offset and the tipheight, the interaction includes at least one of adding one or similaror different objects to the objects at the target location, removing theone or more objects of interest at the target location, applying a fluidto the one or more objects of interest, removing fluid applied to theone or more objects of interest, dislodging the one or more objects ofinterest to separate the one or more objects from at least one otherobject at the target location, moving the one or more objects ofinterest at the target location to a location on the stage differentfrom the target location, stamping the one or more objects of interest,stirring a medium in which the one or more objects are disposed in atthe target location, aspirating the one or more objects from the stage,scraping the one or objects of interest.